Skip to content

fix(ocr): recover from Vision failures on macOS 27 - #95

Merged
melonamin merged 3 commits into
mainfrom
fix/vision-failure-recovery
Sep 27, 2026
Merged

melonamin merged 3 commits into
mainfrom
fix/vision-failure-recovery

Conversation

@melonamin

@melonamin melonamin commented Sep 26, 2026 •

Copy link
Copy Markdown
Member

Fixes #92

Root cause

On macOS 27, Vision's .accurate recognition path runs a model through the Neural Engine compiler (E5RT). When compilation fails, handler.perform throws (E5RT Error ... Code: 13); performVisionOCR caught the error, logged it, and returned nil — the capture failed with no feedback.

Reproduced part of this locally on macOS 27.0 (26A428): the first .accurate request in a fresh process stalled for 30.5s while the system compiled the model, then succeeded (cached afterward). That matches the reported 10+ second lockup: performVisionOCR had no timeout, so isCaptureInProgress stayed set for the whole stall and every menu action was silently ignored (beginCaptureTransaction guard). The hard E5RT failure did not reproduce here (compilation succeeds on this machine), but .fast completed in 0.01–0.03s in all runs — it uses a different model path, which is why it's the retry level.

Migration to the modern Vision API (macOS 15+)

@hiyotim's testing in #92 pinned down the failure mode precisely: with the legacy VNRecognizeTextRequest (revision 3), the first request in a process succeeds (~15s), then every subsequent request fails in ~2ms with E5RT Code 13 until the process restarts — reproduced outside TRex with a plain request loop. VNRecognizeTextRequestRevision2 works but lacks Cyrillic; the modern Swift Vision API (RecognizeTextRequest) works consistently, including Russian. The same regression is reported in m-tkg/clipkun#12/#14 and jfarcand/mirroir-mcp#36.

So VisionOCREngine now routes through the modern RecognizeTextRequest API, gated with if #available(macOS 15.0, *):

  • Full-fidelity mapping of the legacy configuration: recognition level (.accurate/.fast), recognitionLanguages (BCP-47 strings converted to Locale.Language), usesLanguageCorrection, automatic language detection, custom words, minimum text height — and identical OCRResult construction (joined top candidates, averaged confidence, level string).
  • Same retry semantics as the legacy path: on failure at .accurate, retry once at .fast with a fresh request.
  • The legacy VNRecognizeTextRequest path (with its .fast retry) remains for macOS 14.x, and everything below stays as the safety net for any remaining failure.
  • No observation/bounding-box consumers are affected: all VNRecognizedTextObservation handling is internal to the engine, and table detection uses the separate RecognizeDocumentsRequest path (already the modern API).

Verified against the reported failure mode on this macOS 27 machine: a new regression test runs 6 sequential .accurate recognitions on rendered images through the migrated path — all succeed with correct text (the legacy loop's failure begins at request 2, per hiyotim's reproduction). A second new test confirms Cyrillic recognition, the capability that ruled out the revision-2 workaround.

Changes

  • VisionOCREngine.recognizeText: on macOS 15+, use the modern RecognizeTextRequest API (immune to the E5RT regression); on macOS 14.x, keep the legacy request. Either way, when recognition throws at .accurate, retry once with a fresh request at .fast before propagating the error. Covers every Vision caller (capture, clipboard, watch mode, automation, LLM fallback).
  • performVisionOCR: each attempt is now bounded by a 10s timeout (same pattern as performOCR's existing 5s timeout), with a .fast retry on timeout or error. A stalled model compile can no longer freeze the pipeline; the in-progress flag always resets (already defer-guarded on all paths, now bounded too).
  • If Vision still fails: fall back to the LLM OCR engine or Tesseract if the user has one configured (skipping an engine that already failed and fell back to Vision), mirroring the existing recognizeImage routing.
  • If nothing recovers: post a "Capture failed" notification (bypasses the result-notification preference; errors shouldn't be silent) or write to stderr in the CLI. Watch mode suppresses the notification and logs instead, since it polls repeatedly.

Verification

  • swift test --package-path Packages/TRexCore: 30 tests pass, including 9 covering fallback engine selection, fallback execution with injected failing/fixed-result OCREngine test doubles, both recognition levels on a rendered image, and in-progress state reset after a failed capture — plus 2 new ones: repeated sequential recognition through the modern path (issue macOS 27 issues #92's exact failure mode) and Cyrillic recognition.
  • xcodebuild -scheme TRex -configuration Debug build CODE_SIGNING_ALLOWED=NO -skipMacroValidation: builds clean.
  • The hard E5RT Code 13 failure does not occur on this machine (a legacy-API request loop succeeds here), so the legacy-path recovery chain is verified through the injected-engine tests; the modern-path immunity hiyotim observed is verified by the repeated-recognition test.

Vision's accurate recognition path can fail or stall at the Neural
Engine layer on macOS 27 (E5RT error code 13), leaving captures
failing silently and the app unresponsive until restart.

- Retry with a fresh request at the fast recognition level when
  handler.perform throws at the accurate level
- Bound each Vision attempt with a timeout so a stalled model
  compilation cannot freeze the capture pipeline
- Fall back to a configured LLM or Tesseract engine when Vision
  still fails, skipping an engine that already failed
- Surface a user-facing notification (stderr for the CLI) instead
  of failing silently; watch mode logs instead to avoid spam
The legacy VNRecognizeTextRequest revision-3 path is broken OS-side on
macOS 27 (issue #92): after a process's first successful request, every
subsequent request fails with E5RT error code 13 until restart. The
modern Swift Vision API is unaffected, so route VisionOCREngine through
RecognizeTextRequest wherever it exists, mapping recognition level,
languages, correction, and custom words onto the new request and keeping
the fast-level retry. The legacy path remains for macOS 14.x, and the
engine-fallback safety net is unchanged.
Review findings on #95: the Vision-failure fallback passed Vision's language list to Tesseract (empty under automatic detection, so Tesseract defaulted to English); it now uses the configured Tesseract languages. The accurate-level retry no longer fires on CancellationError from an abandoned timed-out task. Recognition tests now assert the accurate level actually served the result so the silent fast retry cannot mask the issue #92 regression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS 27 issues

1 participant